Method and apparatus for storing and processing picture information of temporally successive pictures

ABSTRACT

A method and apparatus are provided for storing and processing picture information of temporally successive pictures wherein in each case at least one picture area is selected from each temporally preceding picture taking into account, the picture information determined from each selected picture area is stored with an assignment information item, which enables an assignment of the respective picture area and thus also of the picture information of the respective picture area to the corresponding picture and to the position of the picture area within the corresponding picture, at least one mixed picture is formed from the picture information of the respectively stored, selected picture areas, taking the account of the assignment information, and the mixed picture is used in the context of a motion estimation, a motion compensation or an error masking for an image.

BACKGROUND OF THE INVENTION

Such a method and such a device are disclosed in [1]. This knownprocedure utilizes a motion estimation for picture information frompicture areas of a number of temporally preceding pictures. A number ofpredecessor pictures are used for the motion estimation in order to makeavailable a larger selection of picture areas, thereby improving themotion estimation and, associated with this, a motion compensation or anerror concealment.

This can be explained, for example, by the fact that, in the case ofbackground becoming free in a picture, between the picture that is to becoded or generally to be processed and its directly temporally precedingpicture, a motion estimation is highly erroneous when only these twopictures are taken into account. However, if a number of predecessorpictures are taken into account in the context of the motion estimationor the error concealment, then it is possible to replicate these pictureareas of the background becoming free from earlier, temporally precedingpictures.

Usually, in the case of so-called predictive coding methods, that is tosay in the case of coding methods in which a so-called motion estimationis carried out, the present picture information is estimated fromalready transmitted picture information from picture areas and only aso-called estimation error produced as a result of the motion estimationis communicated. The more accurate this motion estimation, the moreeffectively the sequence of temporally successive pictures (alsoreferred to as picture sequence hereinafter) can be compressed. To date,it has been customary to apply the motion estimation only to thedirectly temporally preceding picture. If objects move against abackground of the picture, then the background becoming free cannot bepredicted from the directly temporally preceding predecessor picture;rather, it must be coded completely anew. As a result, the efficiency ofthe picture compression decreases and the required data rate fortransmission of the picture sequence with constant quality increases.

In the case of the procedure disclosed in [1], all of the predecessorpictures which are taken into account over a number of temporallypreceding pictures in the context of the motion estimation are stored intheir entirety. This leads to a considerable memory space requirement,particularly in applications in which only a relatively small memoryspace is available, for example in video telephony. In the case of thisknown procedure, a multiple of the memory space that is usuallyavailable for picture coding is required to store the necessary pictureinformation of the multiplicity of temporally preceding pictures.

Consequently, the present invention is based on the problem of storingand processing picture information of temporally successive pictures,the quality of a motion estimation or an error concealment being similarto that as in the case of the procedure disclosed in [1], but the memoryspace requirement for storing the multiplicity of temporally precedingpictures being reduced.

SUMMARY OF THE INVENTION

In the case of a method for storing and processing picture informationof temporally successive pictures, according to the present invention,in each case at least one picture area is selected from each temporallypreceding picture taken into account. The picture information determinedfrom each selected picture area is stored with an assignment informationitem, which enables an assignment of the respective picture area and,thus, also of the picture information of the respective picture area tothe corresponding picture and to the position of the picture area withinthe corresponding picture. At least one mixed picture is formed from thepicture information of the respectively stored, selected picture areas,taking account of the assignment information. The mixed picture is usedin the context of a motion estimation or a motion compensation or anerror concealment for a picture.

An inventive device for storing and processing picture information oftemporally successive pictures has a processor which is set up in such away that the following steps can be carried out:

-   -   in each case, at least one picture area is selected from each        temporally preceding picture taken into account,    -   picture information from each selected picture area is stored        with an assignment information item, which enables an assignment        of the respective picture area to the corresponding picture and        to the position of the picture area within the picture,    -   at least one mixed picture is formed from the picture        information from the stored selected picture areas, with the        respective assignment information being taken into account,    -   the mixed picture is used in the context of a motion estimation,        a motion compensation or an error concealment for a picture.

In a computer-readable storage medium of the present invention, acomputer program for storing and processing picture information oftemporally successive pictures is stored, which, if it is executed by aprocessor, has the following method steps:

-   -   in each case, at least one picture area is selected from each        temporally preceding picture taken into account,    -   picture information from each selected picture area is stored        with an assignment information item, which enables an assignment        of the respective picture area to the corresponding picture and        to the position of the picture area within the picture,    -   at least one mixed picture is formed from the picture        information from the stored selected picture areas, with the        respective assignment information being taken into account,    -   the mixed picture is used in the context of a motion estimation,        a motion compensation or an error concealment for a picture.

Moreover, an inventive computer program element for storing andprocessing picture information of temporally successive pictures isdisclosed which has the following method steps, if it is executed by aprocessor:

-   -   in each case, at least one picture area is selected from each        temporally preceding picture taken into account,    -   picture information from each selected picture area is stored        with an assignment information item, which enables an assignment        of the respective picture area to the corresponding picture and        to the position of the picture area within the picture,    -   at least one mixed picture is formed from the picture        information from the stored selected picture areas, with the        respective assignment information being taken into account,    -   the mixed picture is used in the context of a motion estimation,        a motion compensation or an error concealment for a picture.

Since the invention makes it possible no longer to store the temporallypreceding pictures in their entirety, but rather to select and storepicture areas of temporally preceding pictures, a considerable reductionof the necessary memory space requirement is achieved.

Nevertheless, the motion estimation quality that can be achieved remainsessentially identical to the quality which can be achieved with themethod disclosed in [1].

Clearly, the present invention can be seen in the fact that relevantareas for the motion estimation of temporally preceding pictures thatare taken into account are preferably stored, instead of storing theentire picture information of temporally preceding pictures for themotion estimation over a multiplicity of temporally preceding pictures.

The invention can be embodied both by a computer program, that is to sayin software, and by a special electronic circuit in hardware.

In one embodiment of the present invention, it is provided that apicture area is selected when the picture information in thecorresponding picture area is sufficiently suitable with regard to apredetermined criterion for the motion estimation of two temporallysuccessive pictures; that is to say, when the picture informationsatisfies a predetermined criterion.

In a further embodiment of the present invention, it is provided that anumber of successive temporally preceding pictures are used in order toselect and store picture areas from these pictures.

The picture information can be stored in a ring memory structure, thatis to say a memory space of predetermined size can be made available forstoring the picture information of temporally preceding pictures and,for the case where the memory area made available has been filled,picture areas, that is to say the picture information of thecorresponding picture areas, correspondingly can be overwritten by “moreup-to-date” picture information.

This can be done by the picture information which has been stored forthe longest time in the memory being overwritten by the more up-to-datepicture information that is to be newly stored.

As an alternative, it is possible to replace the picture informationwhich has been selected the least often in the context of the motionestimation, motion compensation or the error concealment, since, from aheuristic standpoint, this picture information is not sufficientlysuitable for the motion estimation and, consequently, also need notremain in storage.

This development can ensure a deterministically prescribed memory spacerequirement which is considerably less than the memory space requirementin the case of the procedure disclosed in [1].

A motion estimation can be carried out on the at least one mixed picturewith regard to the picture to be processed.

Furthermore, a number of mixed pictures can be formed from the pictureinformation from picture areas of a number of temporally precedingpictures and, for each mixed picture, a motion estimation can be carriedout with regard to the picture.

In accordance with this embodiment, the mixed picture is selected forwhich the motion estimation yields a best result with regard to apredetermined criterion. The motion vectors are formed for the pictureareas of the selected mixed picture, which motion vectors finally can beused in the context of the motion compensation or the error concealment.

The mixed pictures can be formed iteratively for each temporallypreceding picture taken into account progressively proceeding from thepicture for which further processing is provided.

This can be done by the stored picture information of a respectivetemporally preceding picture being copied into a mixed picture and theresulting mixed picture being stored. The stored picture information ofa respective temporally preceding picture is copied into the mixedpicture resulting in the preceding step. The method is continuedprogressively for each mixed picture to be formed until all thetemporally preceding pictures have been taken into account.

The picture area to be selected generally can have any desired form, butthe picture object preferably has a predeterminable form or is amacroblock or a picture block.

This development makes it possible to readily insert the method intocustomary block-based and/or object-based picture coding methods.

The present invention can be used both in the context of video telephonyand in digital television or in so-called HDTV. Furthermore, the presentinvention advantageously can be used in Internet video streaming, in thearea of video on demand, in business television, or in Internettelevision.

It should be pointed out that the present invention can be used in thecontext of any picture coding method in which a motion estimation iscarried out; that is to say, in any predictive picture coding method.

A further refinement provides for the picture information from eachselected picture area and/or the associated assignment information to becompressed prior to storage.

Additional features and advantages of the present invention aredescribed in, and will be apparent from, the following DetailedDescription of the Invention and the Figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows a sketch illustrating the principle of the procedure inaccordance with an exemplary embodiment of the present invention.

FIG. 2 shows a coding arrangement in accordance with an exemplaryembodiment of the present invention.

FIG. 3 shows a flow diagram illustrating the individual method steps ofthe method in accordance with an exemplary embodiment of the presentinvention.

FIG. 4 shows a decoding arrangement in accordance with an exemplaryembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 2 shows a coding arrangement 200 with a camera 201, which recordspictures of a user 202 of the coding arrangement 200. The camera 201 isconnected to a source coder 204 via a line 203. In the source coder 204,the video pictures that have been recorded by the camera and digitizedare source-coded in accordance with the procedure described below.

The individual video pictures have pixels which are assigned codinginformation, for example color information or brightness information.The pixels are grouped into picture blocks 205, which are, in turn,grouped into so-called macroblocks.

Picture blocks 205 of a picture to be coded are fed to the source coder204. In the source coder 204, the picture blocks 205 are fed to a unitfor motion estimation 206 and to a switch unit 207. The switch unit 207is used to determine, under the control of a control unit (notillustrated), whether a motion estimation is carried out for the pictureblock and, consequently, a difference picture information item 208between coding information of the picture block to be coded and apicture block of a temporally preceding picture is to be generated, orwhether the complete coding information 208 of the picture block 205 tobe coded is to be fed via the switch unit 207 to a transformation unit209.

In the transformation unit 209, a Discrete Cosine Transform (DCT) iscarried out on the coding information to be coded or the differencecoding information, thereby forming transform coefficients 210 which arequantized to form quantized transform coefficients 212 in a quantizationunit 211.

The quantized transform coefficients 212 are fed to a unit for runlength coding 213. The run-length-coded transform coefficients 214 arewritten to an output buffer 215. The run-length-coded transformcoefficients 214 are read from the output buffer 215 and fed, forexample, to a channel coding and transmitted to a receiver.

The quantized transform coefficients 212 are furthermore fed in afeedback loop to an inverse quantization unit 216, where they aresubjected to inverse quantization to form inverse-quantized transformcoefficients 217.

The inverse-quantized transform coefficients 217 are fed to an inversetransformation unit 218, where the inverse-quantized transformcoefficients 217 are subjected to an Inverse Discrete Cosine Transform(IDCT), thereby forming inverse-transformed coefficients 219 which arestored in a first memory 220.

In the unit for motion estimation 206, a motion estimation is carriedout using the inverse-transformed coefficients 219 stored in the firstmemory 220.

After motion estimation has been carried out, in accordance with thisexemplary embodiment a decision is made as to whether or not a pictureblock is well suited to a motion estimation.

This is done, as is described below, by analysis of a multiplicity ofpictures and checking whether a respective picture block is suitable fora motion estimation; that is to say, can cause a statistically low errorvalue in the context of the motion estimation and can thus statisticallyensure a low data rate.

This is the case, for example, for picture blocks which describe abackground becoming free, that is to say picture blocks which describean object which, between temporally successive pictures, varies greatlywith regard to its position within the picture sequence. The selectedpicture blocks, that is to say the picture information of the selectedpicture blocks, are stored in a second memory 222.

This procedure is explained in more detail below with reference to FIG.1.

The motion vectors 223 resulting in the context of the motion estimationare fed to the unit for run length coding 213, where they are subjectedto run length coding and are processed further.

FIG. 1 shows a sequence 100 of digitized pictures 101, 102, 103, 104,105. In accordance with this exemplary embodiment, F(n) designates thepresent picture to be coded, for which a motion estimation is to becarried out. The preceding picture the furthest back in time which istaken into account in the context of the method described below isdesignated by F(n−k). This means that k temporally preceding picturesare taken into account in the context of the method for motionestimation.

The pictures have picture blocks 106. Selected picture blocks areidentified by the reference symbols 107. Coding information of theselected picture blocks 107 are stored in the second memory 222 togetherwith an assignment information item. The assignment information itemcontains a specification concerning the picture in which the selectedpicture block is situated, and a coordinate specification whichunambiguously describes the position of the selected picture blockwithin the corresponding picture.

The assignment information item is designated by B(i, x, y), where idesignates the respective picture in which the selected picture block issituated, x designates an x-coordinate specification which specifies theposition with regard to the x-coordinate within the selected pictureblock within the corresponding temporally preceding picture, and

y designates a y-coordinate specification which specifies the positionwith regard to the y-coordinate within the selected picture block withinthe corresponding temporally preceding picture.

The second memory 222 has a ring memory structure, as is indicated inFIG. 1.

If the unit for motion estimation 207 carries out a motion estimation,then this is done in the manner illustrated in FIG. 3, within a searcharea 108.

FIG. 3 shows the individual method steps of the method for motionestimation 300 in accordance with this exemplary embodiment.

The method is started in a first step (step 301).

In a further step, for the search area 108, an initialization mixedpicture 109 is formed from coding information of picture blocksexclusively of the temporally directly preceding picture (step 302).This information is available in its entirety, thereby ensuring thatcoding information is available in the entire search area for the motionestimation.

For the corresponding picture block to be coded, a motion estimation iscarried out (step 303) within the search area 108 in the initializationmixed picture 109, thereby forming a first motion vector. Moreover, theresult of the motion estimation is an error value specifying the extentto which the coding information items in the selected area within thesearch area in the initialization mixed picture 302 differ from that ofthe coding information in the picture block to be coded. This can bedone, for example, by forming the sum of the square differences of theindividual coding information items of the pixels taken into account.

The error value and the associated motion vector are stored (step 304).

In a further step (step 305), a second mixed picture 109 is formed bythe selected picture blocks of the temporally nearest preceding pictureF(n−1) which are situated in the selected search area 108 being copiedinto the initialization mixed picture 109 and overwriting the codinginformation of the picture blocks of the initialization mixed picture atthe corresponding position.

A motion estimation is then carried out (step 306) on the further mixedpicture again with regard to the picture to be coded and a second motionvector and a second error value are formed, which are again stored (step307).

In a checking step (step 308), a check is made to determine whether atermination criterion is satisfied.

The termination criterion in accordance with this exemplary embodimentis whether the error value or the second error value is less than apredetermined threshold value.

If neither the error value nor the second error value is less than thepredetermined threshold value, then a third mixed picture is formed inthe same way as the further mixed picture, iteratively by overwritingthe selected picture blocks of the temporally third preceding picture,whose picture blocks are situated in the search area.

Generally, in each case, if a check is made to determine whether thetermination criterion is satisfied and it emerges that the terminationcriterion is not satisfied, a mixed picture of a further iteration isformed by overwriting coding information of a temporally preceding mixedpicture by coding information from picture blocks of a further precedingpicture further back in time, whose picture blocks are situated in thesearch area, and in each case a motion estimation is again carried outon the mixed picture of the next iteration, as a result of which amotion vector of the respective next iteration and a further error valueof the next iteration are formed and the checking step (step 308) isagain applied, now in particular to the error value of the nextiteration. If the termination criterion is satisfied, then thecorresponding further motion vector formed and the specification of theiteration of the corresponding mixed picture in which this motion vectorwas determined are fed to the unit for run length coding 213.

Furthermore, the coding information of the corresponding selectedpicture area in the search area is fed as selected picture block fordifference formation to the switch unit 207 (step 309).

If the predetermined memory area of the second memory 222 is full, thenfurther coding information items of further, more up-to-date selectedpicture blocks, the run-length-coded motion vectors and thecorresponding assignment information are likewise subjected to runlength decoding (not illustrated) and stored in the first memory 220,thereby overwriting stored coding information items which have not beenused the longest in the context of the motion estimation.

The run-length-coded motion vectors, mixed picture indices and therun-length-coded transform coefficients are received by a receiver,where they are fed to a unit for decoding 400 (cf FIG. 4).

The run-length-coded transform coefficients 214 are fed to a unit forthe inverse quantization unit 401, whereby inverse-quantized transformcoefficients 402 are formed.

The inverse-quantized transform coefficients 402 are fed to an inversetransformation unit 403 in which, by carrying out an inverse discretecosine transform, inverse-transformed coefficients 404 are formed andare stored in a first memory 405.

Using the motion vectors determined and the reconstructed pictures inthe context of decoding, the corresponding required mixed picture is, ineach case, stored in a second memory 406.

In a unit for motion compensation 407, the corresponding motion vectorand the mixed picture are used to effect a reconstruction of therequired picture block, that is to say the coding information of thecorresponding picture block, which is fed as difference information 408to an addition unit 409 and added in accordance with the reconstructeddifference coding information.

In this way, the reconstructed picture is generated by a sequence ofreconstructed picture blocks.

A number of alternatives to the exemplary embodiment illustrated aboveare described below.

Various possibilities are provided for selecting the relevant pictureblocks which are stored in the second memory 222. Thus, it is possibleto select those picture blocks whose coding information has changedsignificantly compared with the corresponding picture block of thepreceding picture; that is to say, whose coding information differenceis greater than a predetermined second threshold value.

A further selection criterion may be that so-called intra-coded pictureblocks are selected; that is to say, picture blocks whose codinginformation has been completely coded, and not just as differencepicture blocks.

This procedure is suitable, in particular, for the case where the entirepicture is not intra-coded, but rather only parts of the picture.

The present invention is likewise suitable for an error concealment;that is to say, for an error concealment in the coder device.

In this alternative embodiment, in accordance with the methodillustrated in FIG. 3, the received picture is reconstructed withcorresponding mixed pictures and, for an erroneous picture block forwhich the way in which it is to be decoded is not known, a correspondingpicture block is selected from one of the mixed pictures of temporallypreceding pictures and written to the corresponding erroneous picturearea.

Furthermore, it is possible for the second memory to be preoccupied bytypical picture patterns, which essentially equates to a vectorquantization. What can be used as such picture patterns are, forexample, areal, skin-colored picture blocks or else open or closed eyes,which clearly serve as a code book in the context of a vectorquantization.

Although the present invention has been described with reference tospecific embodiments, those of skill in the art will recognize thatchanges may be made thereto without departing from the spirit and scopeof the present invention as set forth in the hereafter appended claims.

The following publication is cited in this document:

[1] ITU-T H.263+, Annex N

1. A method for storing and processing picture information of temporallysuccessive pictures, the method comprising the steps of: selectingpicture areas within one of the pictures and at least one otherpreceding picture of the temporal succession, said picture areas being aportion of said picture and the at least one preceding picture; copyingand storing the selected picture areas and picture information relatedto each selected picture area, wherein the picture information comprisesan assignment information item that identifies the picture to which eachpicture area is assigned to, and further identifies the coordinatelocation of each picture area within a respectively assigned picture inthe temporal succession; forming at least one mixed picture from thestored picture areas and picture information by processing each picturearea according to the assignment information identified for the pictureareas in the temporal succession; and using the mixed picture for one ofa reference picture for a motion estimation, a reference picture for amotion compensation and an error concealment for a picture.
 2. A methodfor storing and processing picture information of temporally successivepictures as claimed in claim 1, wherein a picture area is selected ifthe picture information in the corresponding picture area satisfies apredetermined criterion for motion estimation of two temporallysuccessive pictures.
 3. A method for storing and processing pictureinformation of temporally successive pictures as claimed in claim 1,wherein picture areas are selected from a plurality of successivetemporally preceding pictures and are stored.
 4. A method for storingand processing picture information of temporally successive pictures asclaimed in claim 1, wherein the picture information is stored in a ringmemory structure.
 5. A method for storing and processing pictureinformation of temporally successive pictures as claimed in claim 1, themethod further comprising the step of carrying out a motion estimationon the at least one mixed picture with regard to the picture.
 6. Amethod for storing and processing picture information of temporallysuccessive pictures as claimed in claim 1, the method further comprisingthe steps of: forming a plurality of mixed pictures from the pictureinformation from picture areas of a plurality of temporally precedingpictures; carrying out, for each mixed picture, a motion estimation withregard to the picture; selecting the mixed picture for which the motionestimation yields a best result with regard to a predeterminedcriterion; forming motion vectors for the picture areas of the selectedmixed picture.
 7. A method for storing and processing pictureinformation of temporally successive pictures as claimed in claim 6,wherein the mixed pictures are formed iteratively for each temporallypreceding picture taken into account progressively proceeding from thepicture for which further processing is provided.
 8. A method forstoring and processing picture information of temporally successivepictures as claimed in claim 6, wherein the mixed pictures are formedprogressively via the steps of: copying the stored picture informationof a respective temporally preceding picture into a mixed picture;storing the resulting mixed picture; copying the stored pictureinformation of a respective temporally further preceding picture intothe mixed picture resulting in the preceding step; and progressivelyrepeating the steps for each mixed picture to be formed until all thetemporally preceding pictures have been taken into account.
 9. A methodfor storing and processing picture information of temporally successivepictures as claimed in claim 1, wherein the picture area is at least oneof a picture object of predeterminable form, a macroblock and a pictureblock.
 10. A method for storing and processing picture information oftemporally successive pictures as claimed in claim 1, wherein the methodis used in one of video telephony, digital television, HDTV, Internetvideo streaming, video on demand, business television, and Internettelevision.
 11. A method for storing and processing picture informationof temporally successive pictures as claimed in claim 1, wherein atleast one of the picture information from each selected picture area andthe associated assignment information is compressed prior to storage.12. A computer-readable storage medium, encoded with a computer programincluding executable instructions, for storing and processing pictureinformation of temporally successive pictures, which, upon beingexecuted by a processor, comprises the steps of: selecting picture areasfor each of at least two temporally preceding pictures; copying andstoring the selected picture areas, along with picture information ofthe selected picture areas with assignment information items related toeach picture area, which enable an assignment of each respective picturearea to the corresponding picture and to a position of the picture areawithin the picture by identifying the picture to which each picture areais assigned to, and further identifying the coordinate location of eachpicture area within a respectively assigned picture in the temporalsuccession; forming at least one mixed picture from the stored pictureareas and picture information, wherein the mixed picture is formed usingthe assignment information items; and using the mixed picture for one ofa reference picture for a motion estimation, a reference picture for amotion compensation and an error concealment for a picture.
 13. Acomputer-readable storage medium, encoded with a computer programincluding executable instructions, for storing and processing pictureinformation of temporally successive pictures, upon being executed by aprocessor, comprises the steps of: selecting picture areas within one ofthe pictures and at least one other preceding picture of the temporalsuccession, said picture areas being a portion of said picture and atleast one preceding picture; copying and storing the selected pictureareas and picture information related to each selected picture area,wherein the picture information comprises an assignment information itemthat identifies the picture to which each picture area is assigned to,and further identifies the coordinate location of each picture areawithin a respectively assigned picture in the temporal succession;forming at least one mixed picture from the stored picture areas andpicture information by processing each picture area according to theassignment information identified for each picture area in the temporalsuccession; and using the mixed picture for one of a reference picturefor a motion estimation, a reference picture for a motion compensationand an error concealment for a picture.
 14. An apparatus for storing andprocessing picture information of temporally successive pictures,comprising: a processor; means for selecting picture areas within one ofthe pictures and at least one other preceding picture of the temporalsuccession, said picture areas being a portion of said picture and atleast one preceding picture; means for copying and storing the selectedpicture areas and picture information related to each selected picturearea, wherein the picture information comprises an assignmentinformation item that identifies the picture to which each picture areais assigned to, and further identifies the coordinate location of eachpicture area within a respectively assigned picture in the temporalsuccession; means for forming at least one mixed picture from the storedpicture areas and picture information by processing each picture areaaccording to the assignment information identified for each picture areain the temporal succession; and means for using the mixed picture forone of a reference picture for a motion estimation, a reference picturefor a motion compensation and an error concealment for a picture.